System and method for cognitive intervention on human interactions

ABSTRACT

A determination regarding whether to intervene in a dialog to provide system-initiated assistive information involves monitoring a dialog between at least two participants and capturing data from a dialog environment containing at least one of the participants. The captured data represent the content of the dialog and physiological data for one or more participants. Assistive information relevant to the dialog and participants is identified, and the captured data are used to determine an intervention index of delivering the assistive information to one or more participants during the dialog. This intervention index is then used to determine whether or not to intervene in the dialog to deliver the assistive information to one or more participants.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of, and claims the benefit ofpriority of, U.S. patent application Ser. No. 15/398,029, filed Jan. 4,2017, titled “System and Method for Cognitive Intervention on HumanInteractions” the entire contents of which are incorporated herein byreference.

FIELD OF THE INVENTION

The present invention relates to system-generated assistance in dialogsystems.

BACKGROUND OF THE INVENTION

Automated dialog systems are used to provide assistance and informationin a computing environment. Examples of automated dialog systems includepersonal assistant solutions such as Siri from Apple, Inc., Google Nowand Microsoft Cortana. These personal assistant solutions returninformation to a user promptly upon request using voice-oriented dialogtechnology. In this context, particular computational components (orbots) integrated to either a voice-oriented or text-based dialog systemcan be programmed to provide assistance and information upon thedetection of certain situations, conditions or events in thecommunication, e.g., in Slack a preconfigured bot user may show a linkabout ‘agile’ when a user message contains this word. Most voice or textactivated assistance systems provide information based on user-initiatedrequest, i.e., like Apple Siri. Alternatively, requests can besystem-initiated, e.g., Slack bots or combination of system-initiatedand user-initiated, e.g., Google Allo.

For system-initiated requests, unsolicited assistance and information,while valuable, is delivered to users as soon as the system detects aneed and computes such unsolicited assistance and information. Thetraditional systems that generate unsolicited assistance and informationdo not take into account the flow of an ongoing discourse orconversation among involved parties or whether the unsolicitedassistance and information can in fact disrupt the ongoing conversation.Therefore, automated dialog systems are desired that providesystem-initiated assistance and information while limiting the adverseeffects of unsolicited assistance and information.

SUMMARY OF THE INVENTION

Exemplary embodiments are directed to cognitive systems with dialog andsystem-initiated assistance and information generation capabilities andto methods for using these cognitive systems. Exemplary embodimentsinclude a method and apparatus for cognitive dialogs that facilitatesthe delivery of valuable information and data, e.g., information anddata that make a dialog more productive by providing a suggestion oravoiding a conflicting situation, during an ongoing dialog without anexplicit request from a user or dialog participant. A given dialog caninclude two or more users or participants. To minimize undesired effectson an ongoing dialog resulting from unsolicited interventions into theongoing dialog, an intervention index is determined before potentialassistive information or data are communicated to the dialogparticipants. Therefore, potential assistive information or data areidentified by the cognitive system for a given situation within anongoing dialog. Then, the intervention index is determined by computingthe intervention cost which reflects the impact on the ongoing dialog ofintervening in the dialog to communicate the potential assistiveinformation or data versus the value of the potential assistiveinformation or data to the ongoing dialog or to the participants in theongoing dialog. An example of the value of the potential assistiveinformation is critical information.

Exemplary embodiments consider dialog and participant specificparameters in order to determine the intervention index, theintervention cost and the value of the potential assistive information.These dialog and participant specific parameters include, but are notlimited to, the context and dynamics of the dialog, e.g., taking turnsin the dialog or doing interventions based on opportunities providedbetween participant turns in the dialog, user data, e.g., participantprofiles, emotional profile of a participant and historical data fromprevious conversations involving one or more of the participants, andthe available hardware and software resources, e.g., mobile phones,tablet computers, and computer-based or cloud based graphical displaysand environments. Therefore, exemplary embodiments balance the value ofinformation against the desired dialog experience to deliver assistiveinformation in a most efficient and least intrusive way.

Exemplary embodiments can be applied to different types of dialogsincluding dialogs to schedule a business trip and dialogs related to anappointment with a health care professional. These dialogs can involveinteractions or communications between two or more participants. Inthese conversations a lack of access to right resources, a lack ofcontextualized information and unexpressed emotions, among other issues,can compromise the effectiveness and productivity of the communication,potentially leading to sub-optimal decisions. For example, in a dialogto schedule a business trip, the potential assistive information relatesto the forecast of a heavy snow at the destination and on the day of thebusiness trip, and the participants are unaware of this forecast. Indetermining the intervention index, the value of the snow storm forecastinformation is high, i.e., it is necessary to avoid a potentially riskyor hazardous situation. Given the very high value of this information,this high level outweighs the intervention cost on the ongoing dialog.Therefore, the intervention index is determined that the forecast shouldbe delivered immediately despite the disruption it may cause in theongoing dialog. Having decided that the intervention index justifiesintervention, a determination is then made regarding the best deliverymethod. This determination considers the participants to which theinformation is to be delivered and the available delivery mechanisms.Regarding the weather forecast, this information should be delivered toall participants in the dialog. Since the participants are interactingthrough a common environment, e.g., a conference call or videoconference, the audio and video capabilities of this common environmentare identified as a mechanism to communicate the forecast information toall of the participants. The result of the intervention to communicatethe weather forecast is a rescheduling of the business trip to a day orlocation with an improved forecast.

In a dialog relating to an appointment with a health care professional,the dialog in particular relates to an appointment with a physician forthe purposes of evaluation, diagnosis and treatment, e.g., theprescription of medication. The potential assistive information is apotential interaction between a drug to be prescribed to the patient andanother drug that the patient is already taking. The physician isunaware of this interaction. Again, the information on potential adversedrug interactions is highly valuable, and this value outweighs anyadverse effects that intervention could cause on the dialog between thephysician and the patient. The intervention index determination,therefore, indicates that the potential assistive information should becommunicated. Therefore, the valuable assistive information, i.e., ahealth risk, should be delivered immediately despite the disruption itmay cause in the dialog. A determination is then made regarding thedelivery mechanism. The participants to whom the information should bedelivered are limited to the physician. The mechanisms available tocommunicate the potential drug intervention information only to thephysician include a tablet computer that the physician is using to enterdiagnostic information and the prescribed information. The result ofthis intervention is the prescription of an alternative medication.

Exemplary embodiments are directed to method for intervening in a dialogto provide system-initiated assistive information. A dialog between atleast two participants is monitored, and data from a dialog environmentcontaining at least one of the participants are captured. The captureddata include at least one of content of the dialog and physiologicaldata for one or more participants. In one embodiment, data capturedevices disposed in the dialog environment are used to monitor thedialog and to capture data. Suitable data capture devices include, butare not limited to, still cameras, video cameras, microphones, telephoneequipment, telephone conferencing equipment, video conference equipment,telephones, cellular telephones, smartphones, tablet computers,electronic mail applications, text message applications, thermostats,thermometers, motion sensors, physiological sensors, activity sensors,biomedical sensors and combinations thereof.

Assistive information relevant to the dialog and participants isidentified. In one embodiment, the captured data are used to identifyassistive information. For example, the captured data are used in atleast one of static searches and dynamic searches over fixed databasesand continuous data streams. In one embodiment, the captured data areused to identify a physical or emotional condition of at least oneparticipant.

The captured data are used to determine an intervention index fordelivering the assistive information to one or more participants duringthe dialog. In one embodiment, a value of the assistive information tothe dialog and participants in the dialog is determined, and anintervention cost on the dialog caused by intervening in the dialog tocommunicate the assistive information is also determined. The value ofthe assistive information is compared to the intervention cost on thedialog, the system intervenes in the dialog if the value of theassistive information is sufficient to justify the intervention cost onthe dialog.

Intervention into the dialog occurs to deliver the assistive informationto one or more participants based on the intervention index. In oneembodiment, intervention parameters are determined for communicating theassistive information to the one or more participants, and interventioninto the dialog occurs in accordance with the intervention parameters.In order to determine the intervention parameters, target participantsin the dialog to receive the assistive information are identified, andan appropriate time during the dialog for intervening in the dialog thatminimizes interference with the participants is also identified. Inaddition, intervention mechanisms associated with the targetparticipants that can be used to deliver the assistive information tothe target participants are identified. Suitable intervention mechanismsinclude, but are not limited to, speakers, video monitors, lights,haptic systems, kinesthetic systems, graphical user interfaces, smartphones, tablet computers, telephones, cellular telephones andcombinations thereof.

Having intervened in the dialog, post-intervention impacts on the dialogresulting from intervening in the dialog and delivering the assistiveinformation are determined. The post-intervention impacts are used todetermine an intervention index of delivering additional assistiveinformation to one or more participants at a later time during thedialog. Determining post-intervention impacts includes analyzing datacaptured from the dialog and participants after intervening in thedialog to determine the post-intervention impacts of intervening in thedialog to deliver the assistive information. In one embodiment, thepost-intervention impacts are associated with at least one of thedialog, a portion of the dialog, a type of dialog, the assistiveinformation, a category of assistive information, a delivery mechanismused to communicate the assistive information and one or moreparticipants. The post-intervention impacts and associations are storedfor use in determining the intervention index of delivering additionalassistive information to one or more participants at the later timeduring the dialog.

Exemplary embodiments are also directed to a method for intervening in adialog to provide system-initiated assistive information where a dialogbetween at least two participants is monitored and data from a dialogenvironment containing at least one of the participants are captured.The captured data include at least one of content of the dialog andphysiological data for one or more participants. Assistive informationrelevant to the dialog and participants is identified, and the captureddata are used to determine an intervention index for delivering theassistive information to one or more participants during the dialog bydetermining a value of the assistive information to the dialog andparticipants in the dialog and determining an intervention cost on thedialog caused by intervening in the dialog to communicate the assistiveinformation.

In one embodiment, using the captured information to determine theintervention index further includes comparing the value of the assistiveinformation to the intervention cost on the dialog, and intervening inthe dialog occurs if the value of the assistive information issufficient to justify the intervention cost on the dialog. In oneembodiment, determining the value of the assistive information includesassigning a low value to assistive information comprising asystem-generated suggestion, assigning a high value to assistiveinformation comprising a resolution to a conflict occurring among theparticipants during the dialog and assigning a medium value between thelow value and the high value to assistive information comprising ananswer to a participant generated question.

In one embodiment, determining the intervention cost on the dialogincludes assigning a low intervention cost level when intervening in thedialog elicits a positive reaction from one or more participants,assigning a high intervention cost level when intervening in the dialogelicits a negative reaction from one or more participants and assigninga medium intervention cost level between the low intervention cost leveland the high intervention cost level when intervening in the dialogelicits a neutral reaction from one or more participants.

Exemplary embodiments are also directed to a dialog intervention system.The dialog intervention system includes at least one dialog environmentcontaining at least two participants engaged in a dialog and a cognitivesystem in communication with the at least one dialog environment tomonitor the dialog between the at least two participants. The cognitivesystem includes interface capture modules to capture data from thedialog environment. The captured data are at least one of content of thedialog and physiological data for one or more participants. Thecognitive system also includes processing modules to identify assistiveinformation relevant to the dialog and participants and to use thecaptured data to determine an intervention index for delivering theassistive information to one or more participants during the dialog.Interface presentation modules are included in the cognitive system tointervene in the dialog to deliver the assistive information to one ormore participants based on the intervention index.

In one embodiment, data capture devices are located in the dialogenvironment to monitor the dialog and to capture data. Suitable datacapture devices include, but are not limited to, still cameras, videocameras, microphones, telephone equipment, telephone conferencingequipment, video conference equipment, telephones, cellular telephones,smartphones, tablet computers, electronic mail applications, textmessage applications, thermostats, thermometers, motion sensors,physiological sensors, activity sensors, biomedical sensors orcombinations thereof. In one embodiment, the processing modules includean assistive information detection module to use the captured data toidentify assistive information. In one embodiment, the assistiveinformation detection module uses the captured data in at least one ofstatic searches and dynamic searches over fixed databases and continuousdata streams. In one embodiment, the processing modules use the captureddata to identify a physical or emotional condition of at least oneparticipant.

In one embodiment, processing modules include an assistive informationvalue and intervention cost module to determine a value of the assistiveinformation to the dialog and participants in the dialog and todetermine an intervention cost on the dialog caused by intervening inthe dialog to communicate the assistive information. The assistiveinformation value and intervention cost module compares the value of theassistive information to the intervention cost on the dialog, and theinterface modules intervene in the dialog if the value of the assistiveinformation is sufficient to justify the intervention cost on thedialog.

In one embodiment, the processing modules include an interventiondelivery module determine intervention parameters for communicating theassistive information to the one or more participants; and the interfacemodules intervene in the dialog in accordance with the interventionparameters. The intervention delivery module identifies targetparticipants in the dialog to receive the assistive information. Theprocessing modules include an intervention opportunity analysis moduleto identify an appropriate time during the dialog for intervening in thedialog that minimizes interference with the participants, and the dialogenvironment includes intervention mechanisms associated with the targetparticipants and in communication with the interface modules that can beused to deliver the assistive information to the target participants.Suitable intervention mechanisms include, but are not limited to,speakers, video monitors, lights, haptic systems, kinesthetic systems,graphical user interfaces, smart phones, tablet computers, telephones,cellular telephones and combinations thereof.

In one embodiment, the processing modules include an intervention resultanalysis module to determine post-intervention impacts on the dialogresulting from intervening in the dialog and delivering the assistiveinformation and to use the post-intervention impacts to determine anintervention index for delivering additional assistive information toone or more participants at a later time during the dialog. In oneembodiment, the intervention result analysis module analyzes datacaptured from the dialog and participants after intervening in thedialog to determine the post-intervention impacts of intervening in thedialog to deliver the assistive information. In another embodiment, theintervention result analysis module associates the post-interventionimpacts with at least one of the dialog, a portion of the dialog, a typeof dialog, the assistive information, a category of assistiveinformation, a delivery mechanism used to communicate the assistiveinformation and one or more participants and stores thepost-intervention impacts and associations for use in determining theintervention index for delivering additional assistive information toone or more participants at the later time during the dialog.

Exemplary embodiments are also directed to a dialog intervention systemthat contains at least one dialog environment containing at least twoparticipants engaged in a dialog and a cognitive system in communicationwith the at least one dialog environment to monitor the dialog betweenthe at least two participants. The cognitive system includes interfacecapture modules to capture data from a dialog environment containing atleast one of the participants. The captured data are at least one ofcontent of the dialog and physiological data for one or moreparticipants. The cognitive system also includes processing modules incommunication with the interface capture modules. The processing modulesinclude an assistive information detection module to identifyingassistive information relevant to the dialog and participants and anassistive information value and intervention cost analysis module to usethe captured data to determine a value of the assistive information tothe dialog and participants in the dialog and to determine anintervention cost on the dialog caused by intervening in the dialog tocommunicate the assistive information in order to determine anintervention index for delivering the assistive information to one ormore participants during the dialog.

In one embodiment, the assistive information value and intervention costanalysis module compares the value of the assistive information to theintervention cost on the dialog, and the cognitive system includesinterface presentation modules to intervene in the dialog if the valueof the assistive information is sufficient to justify the interventioncost on the dialog. In one embodiment, the assistive information valueand intervention cost analysis module assigns a low value to assistiveinformation comprising a system-generated suggestion, assigns a highvalue to assistive information comprising a resolution to a conflictoccurring among the participants during the dialog and assigns a mediumvalue between the low value and the high value to assistive informationcomprising an answer to a participant generated question. In oneembodiment, the assistive information value and intervention costanalysis module assigns a low intervention cost level when interveningin the dialog elicits a positive reaction from one or more participants,assigns a high intervention cost level when intervening in the dialogelicits a negative reaction from one or more participants and assigns amedium intervention cost level between the low intervention cost leveland the high intervention cost level when intervening in the dialogelicits a neutral reaction from one or more participants.

In one embodiment, the processing modules include an intervention resultanalysis module to determine post-intervention impacts on the dialogresulting from intervening in the dialog and delivering the assistiveinformation and to use the post-intervention impacts to determine anintervention index for delivering additional assistive information toone or more participants at a later time during the dialog. In oneembodiment, the intervention result analysis module associates thepost-intervention impacts with at least one of the dialog, a portion ofthe dialog, a type of dialog, the assistive information, a category ofassistive information, a delivery mechanism used to communicate theassistive information and one or more participants and stores thepost-intervention impacts and associations for use in determining theintervention index for delivering additional assistive information toone or more participants at the later time during the dialog.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic representation of an embodiment of a cognitivesystem for use in intervening in a dialog to provide system-initiatedassistive information;

FIG. 2 is a flow chart illustrating an embodiment of a method forintervening in a dialog to provide system-initiated assistiveinformation;

FIG. 3 is a schematic representation of an embodiment of assistiveinformation values and intervention cost levels;

FIG. 4 depicts a cloud computing environment according to an embodimentof the present invention; and

FIG. 5 depicts abstraction model layers according to an embodiment ofthe present invention.

DETAILED DESCRIPTION

Exemplary embodiments implement a cognitive system or cognitive entityas a passive or monitoring participant in a dialog between two or moreparticipants. The cognitive system monitors information andcommunications exchange among the active participants in the dialog toidentify the need for relevant and valuable assistive information. Thecognitive system then determines both when and how to deliver thisassistive information to the appropriate participant or participants inthe least intrusive manner. Monitoring of ongoing dialogs includes, butis not limited to, monitoring audio and visual information and textexchanged among the participants. The exchanged audio, visual and textinclude verbal communications, video, pictures, text messages and sensordata. In general, any information that can capture accurately thecontext of the dialog, e.g., recognize participants through facialrecognition technology, identify emotions of participants, identifybehaviors of participants and identify environmental conditions, ismonitored.

The assistive information is system-identified and is not explicitlyrequested by an active participant or pre-determined by the system.Instead, the assistive information is identified on-the-fly in real-timebased on the monitored context of the dialog and the participants inthat dialog. Therefore, the identified assistive information is valuableinformation within the context of the ongoing dialog. To minimizeintervention cost associated with interrupting a dialog by communicatingthe assistive information, the intervention index on the interactionflow among the participants is determined. The decision on whether andwhen to present the assistive information to one or more participants inthe dialog is based on the determined intervention index, which is basedon a balance or trade-off between the value of the assistive informationto the dialog or the participants in the dialog and the interventioncost on the dialog of intervening to communicate the assistiveinformation. When the determined intervention index indicates thatintervention is preferred, then a determination is made regarding whichparticipants are to receive the assistive information and the mechanismto deliver the assistive information. The mechanism to deliver theassistive information considers the available mechanisms to communicatethe assistive information to the targeted participants, selecting thebest delivery mechanism. The assistive information is then delivered tothe targeted participants at the most appropriate moment using theselected delivery mechanism. The result of the intervention is thenmonitored and assessed. Future interventions can take into account theresults of previous interventions.

Exemplary embodiments analyze the trade-off between the benefit of theassistive information versus the cost of disruption in the communicationflow, i.e., the intervention cost. In addition, context analysisrelative to the dialog is used to identify the best delivery mechanismand the best moment to communicate the assistive information to anysubset of targeted participants. Embodiments of the cognitive system canbe configured as a stand-alone dialog system with system-initiatedassistance. Alternatively, the cognitive system is integrated into othercognitive systems such as IBM Watson, Apple Ski, Google Now, Amazon Echoand Microsoft Cortana. This integration supports and enhances thecommunication among human agents in a shared space, a distributedcomputing system or a cloud based computing environment.

Exemplary embodiments provided include dialog environments containinginstrumentation and equipment such as microphones and cameras to monitorconversations and reactions by capturing audio and video streams. In oneembodiment, text is captured from a text based input/output device suchas a touch screen or keyboard. The cognitive system monitors the dialogenvironment using the available data capture mechanisms to obtain datafor the ongoing dialog. The dialog environment also includes thedelivery mechanisms to deliver the assistive information to one or moredialog participants. These delivery mechanisms include, but are notlimited to, graphical user interfaces, video monitors, speakers,cellular phones, smartphones, tablet computers, telephone equipmentincluding conferencing equipment and public address systems.

Exemplary embodiments of the cognitive system include computing systemsand modules that provide the analytical and computational components toanalyze the dialog, identify assistive information, determine theintervention index, predict the most appropriate moment to make theintervention and how to deliver the assistive information to any subsetof targeted participants. Suitable computing systems include, but arenot limited to, distributed computing systems and cloud based computingsystems that can be deployed on a cloud platform. These analytical andcomputational components process the data from the monitored dialogenvironment and generate an intervention to deliver assistiveinformation that is relevant for the ongoing dialog. In addition, theanalytical and computational components analyze the context of thedialog to determine if an intervention should be conducted, when theintervention should be conducted and how to deliver and conduct theintervention.

Exemplary embodiments include a method for providing cognitivesystem-initiated assistive information and data to a dialog between atleast two participants. The dialogs include any setting in whichinformation is exchanged through any available mechanism between two ormore active participants. The active participants include humanparticipants and automated or machine participants. Suitable dialogsinclude, but are not limited to, voice communications, conversations,video conferences, live and in-person meetings, text message exchanges,telephone conferences, lectures, debates, television broadcasts,customer service calls, technical support sessions and combinationsthereof. Each location of one or more participants in the dialog is adialog environment.

The dialog between the two or more participants is monitored by thecognitive system. In one embodiment, data capture devices in at leastone dialog environment containing at least one participant are used tomonitor the dialog exchanges and the dialog participants during thedialog. Alternatively, data capture devices in a plurality of dialogenvironments or for a plurality of participants in one or more dialogenvironments are used. Suitable data capture devices include, but arenot limited to, still cameras, video cameras, microphones, telephoneequipment, telephone conferencing equipment, video conference equipment,telephones, cellular telephones, smartphones, table computers,electronic mail applications, text message applications, thermostats,thermometers, motion sensors, physiological sensors, activity sensors,biomedical sensors and combinations thereof. The data capture devicescan be used in combination with software applications that can processthe monitored data such as facial recognition applications, voicerecognition applications and optical character recognition applications.

The data capture devices are used to obtain or capture data andinformation related to at least one of the content or context of thedialog and the status or physical condition of the participants. Thestatus and physical condition of the participants covers the emotionalstate of the participants, e.g., angered, nervous, agitated, and thefocus of the participants, e.g., distracted, fatigued, bored. Featuresfrom the captured data are extracted to identify potential opportunitiesfor assistive information intervention. In addition, the captured data,e.g., audio, video and text data, are analyzed to identify potentialassistive information and data that are relevant to the dialog and theparticipants in the dialog. For example, the captured data can be usedto conduct static searches and dynamic searches over fixed databases andcontinuous data streams. These searches can cover any source ofinformation including databases and web-sites. In addition, the capturedata can be used to derive the status or physical condition of theparticipants. For example, emotion can be derived from audio or videodata or from biomedical sensors such as galvanic skin response sensors.

Having identified relevant assistive data for the monitored, adetermination is made regarding whether or not to intervene into thedialog to provide the assistive information. In order to make thisdetermination, an intervention index is evaluated. The interventionindex analyzes a trade-off or balance between the value of the assistiveinformation to the dialog and each of the participants in the dialog andthe cost of an intervention on the dialog. Therefore, the interventioncost on the operation, content or progress of the dialog is determined.In addition, the value of the assistive information to the dialog or toone or more participants in the dialog is determined. In one embodiment,profiles and preferences of one or more of the participants are alsotaken into consideration when determining at least one of theintervention cost on the dialog and the value of the assistiveinformation. In analyzing the tradeoff, assistive information havinggreater value can balance a higher level of intervention cost on thedialog, while assistive information having lower value will not beshared unless the intervention cost on the dialog is sufficiently low.In one embodiment, a quantitative acceptable intervention indexthreshold is identified, and a score is associated with the level ofintervention cost on the dialog and value of the assistive information.This score is then compared to the acceptable intervention indexthreshold, and assistive information having a score that is less than orequal to the acceptable intervention index threshold are identified forintervention.

If the assistive information is identified for intervention based on theintervention index, then the intervention parameters for the assistiveinformation or data are identified. The intervention parameters includean identification of the participant or participants to receive theassistive information, an identification of the delivery mechanism usedto communicate the assistive information and an appropriate time tointervene in the dialog to deliver the assistive information. Ingeneral, the intervention parameters are selected to minimizeintrusiveness and disruption to the dialog, to limit the distribution ofthe assistive information to those participants for which it isnecessary or desired, and to maximize the efficacy of the assistiveinformation being delivery to the desired participants. In oneembodiment, available delivery mechanisms are identified. Preferably,delivery mechanisms are identified that are associated with theparticipants targeted to receive the assistive information. Suitabledelivery mechanisms include, but are limited to, speakers, videomonitors, lights, haptic or kinesthetic systems, graphical userinterfaces, smart phones, tablet computers, telephones includingcellular telephones and combinations thereof.

Having identified the intervention parameters, intervention into thedialog is accomplished by delivering the assistive information to thetargeted participants using the selected delivery mechanisms at theappropriate time. Monitoring of the dialog and capturing of data relatedto the content of the dialog and condition of the participants continuesafter the assistive information is delivered. Data capture during thecontinued monitoring is analyzed to determine that post-interventionimpacts or results of the intervention into the dialog and delivery ofthe assistive information. The post-intervention impacts look for actualchanges in the dialog or participants in response to the deliveredassistive information, changes in the content or progress of the dialogand changes in the condition or status of the participants. Thesepost-intervention impacts can be associated with the dialog or any partof it, with the type of dialog, with the assistive information or typeof assistive information, with the delivery mechanisms and with theparticipants in the dialog. The post-intervention impacts and anyassociations are then stored to be used in subsequent interventiondeterminations, intervention index determinations and interventionparameter determinations.

Referring initially to FIG. 1, exemplary embodiments are directed to adialog intervention system 100 for intervening in dialogs to providesystem-initiated assistive information to the participants in thedialogs. The dialog intervention system includes a cognitive system 102that is in communication with a dialog being conducted between two ormore participants 108 located in one of more dialog environments 106. Agiven dialog can include two or more participants located at one or moredialog environments. Each dialog environment includes a plurality ofdata capture devices such as video cameras 112, microphones 116,computers 118 and smart phones 114 or text message enabled devices. Eachdialog environment also includes a plurality of intervention mechanismssuch as smart phones 114, speakers 120, computers 118 and video monitors122. Some devices in the dialog environment can function as both datacapture devices and intervention mechanisms. Each dialog environment andin particular the data capture devices and intervention mechanisms arein communication with the cognitive system through one of more local orwide area networks 104.

The cognitive system can be executing one or more computing systemsincluding distributed computing systems and cloud based computingsystems. The cognitive system includes at least one logical processorexecuting software programs stored in databases on one or more hardwarememories. Therefore, the cognitive system includes a plurality ofmodules that provide the desired functions of the cognitive system.These modules include interface modules 126 to monitor and capture datafrom the dialog environments containing the participants. Theseinterface modules are interface capture modules including a text capturemodule 132 for capturing text based information, an audio capture module130 for capturing audio data such as live conversations and a videocapture module 128 for capturing still pictures and video of the dialogenvironment. The interface modules are also interface presentationmodules including a text presentation module 138 for delivering textbased assistive information, an audio presentation module 136 fordelivering audio based assistive information and a video presentationmodule 134 for delivering pictures and video based assistiveinformation.

In addition, the modules include processing modules 124 to analyze thecaptured data, identify relevant assistive information, determinewhether intervention should occur and establish the parameters of thatintervention. All of the processing modules are in communication withthe interface modules. In one embodiment, the interface modules 126 arepart of an instrumented environment, and the processing modules 124 areprovided in a cloud environment. The processing modules include an audioanalysis module 140 to analyze sounds and conversations in the dialogenvironment for dialog context and for emotions or physical status ofthe participants. A video analysis module 142 is provided to analyzepictures, including pictures of participants, objects and text, andvideo from the dialog environment for dialog context and emotions andphysical status of the participants. A dialog or conversation andcontext analysis module 144 receives the analysis of the audio and videoanalysis modules and any other inputs such as text inputs. Theconversation and context analysis module then uses the captured andanalyzed data to determine the context and content of the dialog and toidentify the physical and emotional conditions of the participants. Thecontext and content of the dialog and physical and emotional conditionsof the participants are communicated to an assistive informationdetection module 146, which can use this communicated information tosearch sources of data for assistive information relevant to the dialogand the participants.

An intervention opportunities analysis module 148 also receives thecaptured and analyzed data and follows the dialog to identifyopportunities within the dialog to intervene in the dialog such thatnegative effects on the dialog and participants are minimized. Forexample, the intervention opportunities analysis modules identifyalternating patterns of speech among the participants and breaks withinthe patterns of speech during which an intervention can occur. Anassistive information value and intervention cost module 150 receivesinput from the assistive information detection module and interventionopportunities analysis module. The assistive information andintervention cost module may also have profile and preferenceinformation from the participants and post-intervention impactsinformation from previous interventions. All of this information is usedto determine an intervention index of delivering the assistiveinformation to one or more participants during the dialog. Theintervention index is determined by calculating a value of the assistiveinformation to the dialog and participants in the dialog and anintervention cost on the dialog caused by intervening in the dialog tocommunicate the assistive information. The values can then be comparedto provide the intervention index.

The intervention index is communicated to an intervention deliverymodule 154, which intervenes in the dialog to communicate the assistiveinformation if the intervention index indicates that the value of theassistive information is sufficient to justify the level of interventioncost on the dialog. In addition, the intervention delivery module canuse input from the intervention opportunities analysis module, profilesof the participants and knowledge of the delivery mechanisms in thedialog environments to identify target participants to receive theassistive information, delivery mechanisms associated with the targetparticipants to use in communicating the assistive information and atime to intervene in the dialog to communicate the assistiveinformation. An intervention result analysis module 152 continues tomonitor captured and analyzed data from the dialog following theintervention to determine post-intervention impacts on the dialogresulting from intervening in the dialog and delivering the assistiveinformation. The post-intervention impacts are communicated to otherprocessing modules so that the post-intervention impacts can be used todetermine intervention index of delivering additional assistiveinformation to one or more participants at a later time during thedialog. In one embodiment, the intervention and result analysis moduleassociates the post-intervention impacts with at least one of thedialog, a type of dialog, the assistive information, a category ofassistive information, intervention parameters, and delivery mechanismsused to communicate the assistive information and one or moreparticipants. The post-intervention impacts and generated associationsare then stored for access by other processing modules and determiningthe intervention index of delivering additional assistive information toone or more participants at the later time during the dialog.

Referring now to FIG. 2, exemplary embodiments are also directed to amethod for intervening in a dialog to provide system-initiated assistiveinformation 200. A dialog between at least two participants is monitored202. Suitable dialogs include any communication of audio, video andtextual information among two or more participants. Data from a dialogenvironment containing at least one of the participants are captured.The participants can all be located in a single dialog environment,e.g., a single room, or can be located in a plurality of differentdialog locations. Therefore, each dialog location can include one ormore participants. In addition to physical locations, the dialogenvironment can be virtual locations or virtual environments. Thecaptured data include at least one of content of the dialog andphysiological data for one or more participants.

In one embodiment, data capture devices located in the dialogenvironments are used to monitor the dialog and to capture data 204. Thedialog environments include both physical dialog environments, i.e.,physical locations containing dialog participants and virtual dialogenvironments. Suitable data capture devices include, but are not limitedto, still cameras, video cameras, microphones, telephone equipment,telephone conferencing equipment, video conference equipment,telephones, cellular telephones, smartphones, tablet computers,electronic mail applications, text message applications, thermostats,thermometers, motion sensors, physiological sensors, activity sensors,biomedical sensors and combinations thereof.

Assistive information relevant to the dialog and participants isidentified 206. This information can be general information relevant toany type of dialog or to a specific type of dialog. The assistiveinformation can be obtained in advance and stored. Preferably, theassistive information is obtained in real-time and is specific to thedialog and one or more of the participants in the dialog. Therefore, thecaptured data are used to identify assistive information. In oneembodiment, the captured data are used in at least one of staticsearches and dynamic searches over fixed databases and continuous datastreams. This includes web-based searches and searches of websites andproprietary databases.

While the assistive information can be delivered directly to the dialogand the participants, preferably a determination is made regardingwhether the assistive information should be delivered and the best timeand methods for delivering the assistive information. Therefore, thecaptured data are used to determine an intervention index of deliveringthe assistive information to one or more participants during the dialog208. In one embodiment, the intervention index is determined bycalculating a value of the assistive information to the dialog andparticipants in the dialog 210 and calculating an intervention cost onthe dialog caused by intervening in the dialog to communicate theassistive information 212.

In general, the assistive information value and intervention cost on thedialog are quantified so that they can be compared. For example, theinformation value and intervention cost can be assigned one of aplurality of discrete quantities or levels based on the type ofassistive information and the severity of the intervention cost on thedialog. In one embodiment, three levels are used for the value of theassistive information, and three levels are used for the interventioncost on the dialog. Referring to FIG. 3, the assistive information 301is quantified based on the type 300 of assistive information or thedialog context or event that motivated the system-initiated assistiveinformation to be generated. Three types of assistive information areillustrated; however, more than three types can be used. Each type hasan associated value level. The first type of assistive information isassistive information generated in response to and resolution of aconflict 302 (calendar conflict, weather alert, drug interaction)occurring among participants in the dialog. This type of assistiveinformation is assigned a high value 304. The second type of assistiveinformation is a system-generated suggestion 310 (restaurant, hotel,activity), which is assigned a low value 312. The third type ofassistive information contains answers 306 to participant generatedquestions, whether or not these questions are directed to cognitivesystem, as the cognitive system is a passive monitor of the dialog.Answer-type assistive information is assigned a medium value 308 betweenthe low value and the high value.

The cost on the dialog caused by the intervention can be expressed asintervention costs 314. The same number of discrete intervention costsas the levels of assistive information values can be used.Alternatively, a different number of levels of intervention costs can beused. In one embodiment, three intervention cost levels are used, a lowintervention cost level 316, a high intervention cost level 320 and amedium intervention cost level 318 between the low intervention costlevel and the high intervention cost level. In one embodiment, a lowintervention cost level 316 is used when intervening in the dialogelicits a positive reaction from one or more participants. A highintervention cost level is assigned when intervening in the dialogelicits a negative reaction from one or more participants. The mediumintervention cost level between the low intervention cost level and thehigh intervention cost level is assigned when intervening in the dialogelicits a neutral reaction from one or more participants. In oneembodiment, the captured data can be used to identify a physical oremotional condition of at least one participant. This can be used toidentify participant reactions, value of assistive information andintervention cost on the dialog. The assignment or quantification ofvalues and levels to the assistive information and intervention cost onthe dialog facilitates comparisons 322 for the purpose of determiningthe intervention index.

Returning to FIG. 2, the value of the assistive information is comparedto the intervention cost level on the dialog of intervening on thedialog to communicate the assistive information 214. In order tointervene into the dialog to communicate the assistive information,intervention parameters for communicating the assistive information tothe one or more participants are determined 216. The assistiveinformation is then delivered to the dialog and participants 218 byintervening in the dialog to deliver the assistive information to one ormore participants based on the intervention index. In one embodiment,intervention into the dialog happens if the value of the assistiveinformation is sufficient to justify the intervention cost on thedialog. Intervention into the dialog then occurs in accordance with theintervention parameters.

In order to determine the intervention parameters, target participantsin the dialog to receive the assistive information are identified. Thiscan be all of the participants, a subset of participants or a singleparticipant. In addition, an appropriate time during the dialog forintervening in the dialog is identified that minimizes interference withthe participants. Timing of the intervention takes into account, theflow of the dialog including when each participant is speaking orentering data and pauses or breaks in the dialog or conversation. Forexample, an analysis of speech patterns in a conversation can indicateif a participant is making a statement or asking a question. Thisanalysis can be used to determine when that participant is likely tostop talking before another participant begins talking. In oneembodiment, an interruption objective function is determined for eachparticipant that takes into account not only the relevance and value ofthe assistive information but also turn-taking rules and several otheraspects like culture, hierarchy, relationship of the speakers,participant profiles and emotional states.

Intervention mechanisms associated with the target participants areidentified that can be used to deliver the assistive information to thetarget participants. Suitable intervention mechanisms include, but arenot limited to, speakers, video monitors, lights, haptic systems,kinesthetic systems, graphical user interfaces, smart phones, tabletcomputers, telephones, cellular telephones and combinations thereof.Referring again to FIG. 3, the intervention mechanisms or renderingdevices 324 can be selected based on a combination of the value of theassistive information, the intervention cost level on the dialog and thetarget participants. For example, high value assistive informationjustifies intervention for all intervention cost levels on the dialog.If the intervention cost on the dialog is high or the assistiveinformation is targeted to a single participant, then a less intrusiveintervention mechanism or an intervention mechanism associated with thatsingle participant is utilized, e.g., a table computer 326. For highvalue assistive information having a medium intervention cost on thedialog or targeted to multiple participants, one or more displays 328are used to deliver the assistive information. High value informationwith a low intervention cost on the dialog is delivered using displaysand speakers 330, i.e., devices that can quickly and readily reach thedesired participants. The assistive information values and interventioncost levels on the dialog can also be used to calculate a quantifiedoverall intervention index. This overall intervention index is thencompared to a pre-determined threshold level to determine ifintervention into the dialog to deliver the assistive information isjustified.

Returning again to FIG. 2, monitoring of the dialog and capturing of thedata continues while the dialog continues. Therefore, post-interventionimpacts on the dialog resulting from intervening in the dialog anddelivering the assistive information are determined 220. In oneembodiment, data captured from the dialog and participants afterintervening in the dialog are analyzed to determine thepost-intervention impacts of intervening in the dialog to deliver theassistive information. The post-intervention impacts are associated withat least one of the dialog, a type of dialog, the assistive information,a category of assistive information, a delivery mechanism used tocommunicate the assistive information and one or more participants 222.The post-intervention impacts and associations are stored for use indetermining the intervention index of delivering additional assistiveinformation to one or more participants at the later time during thedialog 224. This facilitates using the post-intervention impacts todetermine intervention index of delivering additional assistiveinformation to one or more participants at a later time during thedialog 226. All of these steps of the method are repeated for theduration of the conversation.

Exemplary embodiments minimize the intrusiveness of the assistiveinformation. The timing and moment of the delivery is based on thetrade-off analysis of the benefit, relevance and value of the assistiveinformation versus the cost of disruption in the communication flowduring the dialog. In one embodiment, the cognitive system is configuredto guarantee the delivery of the assistive information, for example,based on a timeout function when no appropriate time is detected. Inaddition, the assistive information is delivered to the renderingdevices or delivery mechanisms selectively. A context analysis isperformed to determine the best delivery approach of the assistiveinformation to the participants. Some assistive information can bepresented to all participants while other may be appropriated forpresentation to only part of them.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment or an embodiment combining softwareand hardware aspects that may all generally be referred to herein as a“circuit,” “module” or “system.” Furthermore, aspects of the presentinvention may take the form of a computer program product embodied inone or more computer readable medium(s) having computer readable programcode embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention are described above with reference toapparatus (systems) and computer program products according toembodiments of the invention. It will be understood that eachdescription and illustration can be implemented by computer programinstructions. These computer program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the block diagram block orblocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the block diagram block orblocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the block diagram block orblocks.

The schematic illustrations and block diagrams in the Figures illustratethe architecture, functionality, and operation of possibleimplementations of systems, methods and computer program productsaccording to various embodiments of the present invention. In thisregard, each block in the block diagrams may represent a module,segment, or portion of code, which comprises one or more executableinstructions for implementing the specified logical function(s). Itshould also be noted that, in some alternative implementations, thefunctions noted in the block may occur out of the order noted in thefigures. For example, two blocks shown in succession may, in fact, beexecuted substantially concurrently, or the blocks may sometimes beexecuted in the reverse order, depending upon the functionalityinvolved. It will also be noted that each block of the block diagrams,and combinations of blocks in the block diagrams, can be implemented byspecial purpose hardware-based systems that perform the specifiedfunctions or acts, or combinations of special purpose hardware andcomputer instructions.

It is to be understood that although a detailed description on cloudcomputing is provided, implementation of the teachings provided hereinare not limited to a cloud computing environment. Rather, embodiments ofthe present invention are capable of being implemented in conjunctionwith any other type of computing environment now known or laterdeveloped. Cloud computing is a model of service delivery for enablingconvenient, on-demand network access to a shared pool of configurablecomputing resources, e.g., networks, network bandwidth, servers,processing, memory, storage, applications, virtual machines, andservices, that can be rapidly provisioned and released with minimalmanagement effort or interaction with a provider of the service.

This cloud model may include at least five characteristics, at leastthree service models, and at least four deployment models. The fivecharacteristics are on-demand self-service, broad network access,resource pooling, rapid elasticity and measured service. Regardingon-demand self-service, a cloud consumer can unilaterally provisioncomputing capabilities, such as server time and network storage, asneeded automatically without requiring human interaction with theservice's provider. Broad network access refers to capabilities that areavailable over a network and accessed through standard mechanisms thatpromote use by heterogeneous thin or thick client platforms, e.g.,mobile phones, laptops, and PDAs. For resource pooling, the provider'scomputing resources are pooled to serve multiple consumers using amulti-tenant model, with different physical and virtual resourcesdynamically assigned and reassigned according to demand. There is asense of location independence in that the consumer generally has nocontrol or knowledge over the exact location of the provided resourcesbut may be able to specify location at a higher level of abstraction,e.g., country, state, or datacenter. Rapid elasticity refers tocapabilities that can be rapidly and elastically provisioned, in somecases automatically, to quickly scale out and rapidly released toquickly scale in. To the consumer, the capabilities available forprovisioning often appear to be unlimited and can be purchased in anyquantity at any time. For measured service, cloud systems automaticallycontrol and optimize resource use by leveraging a metering capability atsome level of abstraction appropriate to the type of service, e.g.,storage, processing, bandwidth, and active user accounts. Resource usagecan be monitored, controlled, and reported, providing transparency forboth the provider and consumer of the utilized service.

The three service models are Software as a Service (SaaS), Platform as aService (PaaS) and Infrastructure as a Service (IaaS). Software as aservice provides the capability to the consumer to use the provider'sapplications running on a cloud infrastructure. The applications areaccessible from various client devices through a thin client interfacesuch as a web browser, e.g., web-based e-mail. The consumer does notmanage or control the underlying cloud infrastructure including network,servers, operating systems, storage, or even individual applicationcapabilities, with the possible exception of limited user-specificapplication configuration settings. Platform as a service provides thecapability to the consumer to deploy onto the cloud infrastructureconsumer-created or acquired applications created using programminglanguages and tools supported by the provider. The consumer does notmanage or control the underlying cloud infrastructure includingnetworks, servers, operating systems, or storage, but has control overthe deployed applications and possibly application hosting environmentconfigurations. Infrastructure as a service provides the capability tothe consumer to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents, e.g., host firewalls.

The Deployment Models are private cloud, community cloud, public cloudand hybrid cloud. The private cloud infrastructure is operated solelyfor an organization. It may be managed by the organization or a thirdparty and may exist on-premises or off-premises. The community cloudinfrastructure is shared by several organizations and supports aspecific community that has shared concerns, e.g., mission, securityrequirements, policy, and compliance considerations. It may be managedby the organizations or a third party and may exist on-premises oroff-premises. The public cloud infrastructure is made available to thegeneral public or a large industry group and is owned by an organizationselling cloud services. The hybrid cloud infrastructure is a compositionof two or more clouds (private, community, or public) that remain uniqueentities but are bound together by standardized or proprietarytechnology that enables data and application portability, e.g., cloudbursting for load-balancing between clouds.

A cloud computing environment is service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure that includes anetwork of interconnected nodes. Referring now to FIG. 4, anillustrative cloud computing environment 50 is depicted. As shown, thecloud computing environment 50 includes one or more cloud computingnodes 10 with which local computing devices used by cloud consumers,such as, for example, personal digital assistant (PDA) or cellulartelephone 54A, desktop computer 54B, laptop computer 54C, and/orautomobile computer system 54N may communicate. Nodes 10 may communicatewith one another. They may be grouped (not shown) physically orvirtually, in one or more networks, such as Private, Community, Public,or Hybrid clouds as described hereinabove, or a combination thereof.This allows cloud computing environment 50 to offer infrastructure,platforms and/or software as services for which a cloud consumer doesnot need to maintain resources on a local computing device. It isunderstood that the types of computing devices 54A-N shown in FIG. 4 areintended to be illustrative only and that computing nodes 10 and cloudcomputing environment 50 can communicate with any type of computerizeddevice over any type of network and/or network addressable connection,e.g., using a web browser.

Referring now to FIG. 5, a set of functional abstraction layers providedby cloud computing environment 50 (FIG. 4) is shown. It should beunderstood in advance that the components, layers, and functions shownin FIG. 5 are intended to be illustrative only and embodiments of theinvention are not limited thereto. As depicted, the following layers andcorresponding functions are provided. A hardware and software layer 60includes hardware and software components. Examples of hardwarecomponents include: mainframes 61; RISC (Reduced Instruction SetComputer) architecture based servers 62; servers 63; blade servers 64;storage devices 65; and networks and networking components 66. In someembodiments, software components include network application serversoftware 67 and database software 68. A virtualization layer 70 providesan abstraction layer from which the following examples of virtualentities may be provided: virtual servers 71; virtual storage 72;virtual networks 73, including virtual private networks; virtualapplications and operating systems 74; and virtual clients 75.

In one example, management layer 80 may provide the functions describedbelow. Resource provisioning 81 provides dynamic procurement ofcomputing resources and other resources that are utilized to performtasks within the cloud computing environment. Metering and pricing 82provide cost tracking as resources are utilized within the cloudcomputing environment, and billing or invoicing for consumption of theseresources. In one example, these resources may include applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal 83 provides access to the cloud computing environment forconsumers and system administrators. Service level management 84provides cloud computing resource allocation and management such thatrequired service levels are met. Service Level Agreement (SLA) planningand fulfillment 85 provide pre-arrangement for, and procurement of,cloud computing resources for which a future requirement is anticipatedin accordance with an SLA.

Workloads layer 90 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: mapping andnavigation 91; software development and lifecycle management 92; virtualclassroom education delivery 93; data analytics processing 94;transaction processing 95; and method intervening in a dialog to providesystem-initiated assistive information 96.

Methods and systems in accordance with exemplary embodiments of thepresent invention can take the form of an entirely hardware embodiment,an entirely software embodiment or an embodiment containing bothhardware and software elements. In a preferred embodiment, the inventionis implemented in software, which includes but is not limited tofirmware, resident software and microcode. In addition, exemplarymethods and systems can take the form of a computer program productaccessible from a computer-usable or computer-readable medium providingprogram code for use by or in connection with a computer, logicalprocessing unit or any instruction execution system. For the purposes ofthis description, a computer-usable or computer-readable medium can beany apparatus that can contain, store, communicate, propagate, ortransport the program for use by or in connection with the instructionexecution system, apparatus, or device. Suitable computer-usable orcomputer readable mediums include, but are not limited to, electronic,magnetic, optical, electromagnetic, infrared, or semiconductor systems(or apparatuses or devices) or propagation mediums. Examples of acomputer-readable medium include a semiconductor or solid state memory,magnetic tape, a removable computer diskette, a random access memory(RAM), a read-only memory (ROM), a rigid magnetic disk and an opticaldisk. Current examples of optical disks include compact disk-read onlymemory (CD-ROM), compact disk-read/write (CD-R/W) and DVD.

Suitable data processing systems for storing and/or executing programcode include, but are not limited to, at least one processor coupleddirectly or indirectly to memory elements through a system bus. Thememory elements include local memory employed during actual execution ofthe program code, bulk storage, and cache memories, which providetemporary storage of at least some program code in order to reduce thenumber of times code must be retrieved from bulk storage duringexecution. Input/output or I/O devices, including but not limited tokeyboards, displays and pointing devices, can be coupled to the systemeither directly or through intervening I/O controllers. Exemplaryembodiments of the methods and systems in accordance with the presentinvention also include network adapters coupled to the system to enablethe data processing system to become coupled to other data processingsystems or remote printers or storage devices through interveningprivate or public networks. Suitable currently available types ofnetwork adapters include, but are not limited to, modems, cable modems,DSL modems, Ethernet cards and combinations thereof.

In one embodiment, the present invention is directed to amachine-readable or computer-readable medium containing amachine-executable or computer-executable code that when read by amachine or computer causes the machine or computer to perform a methodfor intervening in a dialog to provide system-initiated assistiveinformation in accordance with exemplary embodiments of the presentinvention and to the computer-executable code itself. Themachine-readable or computer-readable code can be any type of code orlanguage capable of being read and executed by the machine or computerand can be expressed in any suitable language or syntax known andavailable in the art including machine languages, assembler languages,higher level languages, object oriented languages and scriptinglanguages. The computer-executable code can be stored on any suitablestorage medium or database, including databases disposed within, incommunication with and accessible by computer networks utilized bysystems in accordance with the present invention and can be executed onany suitable hardware platform as are known and available in the artincluding the control systems used to control the presentations of thepresent invention.

While it is apparent that the illustrative embodiments of the inventiondisclosed herein fulfill the objectives of the present invention, it isappreciated that numerous modifications and other embodiments may bedevised by those skilled in the art. Additionally, feature(s) and/orelement(s) from any embodiment may be used singly or in combination withother embodiment(s) and steps or elements from methods in accordancewith the present invention can be executed or performed in any suitableorder. Therefore, it will be understood that the appended claims areintended to cover all such modifications and embodiments, which wouldcome within the spirit and scope of the present invention.

What is claimed is:
 1. A dialog intervention system comprising: at leastone dialog environment containing at least two participants engaged in adialog; and a cognitive system in communication with the at least onedialog environment to monitor the dialog between the at least twoparticipants, the cognitive system comprising: interface capture modulesto capture data from the dialog environment, the captured dataaccurately capturing a context of the dialog and comprising content ofthe dialog; processing modules to identify assistive informationvaluable within the context of the dialog and to use the captured datato determine an intervention index on an interaction flow among theparticipants from delivering the assistive information to one or moreparticipants during the dialog, wherein the intervention index comprisesa quantification of an intervention cost; and interface presentationmodules to intervene in the dialog to deliver the assistive informationto one or more participants based on the intervention index.
 2. Thesystem of claim 1, further comprising data capture devices disposed inthe dialog environment to monitor the dialog and to capture data.
 3. Thesystem of claim 2, wherein the data capture devices comprise stillcameras, video cameras, microphones, telephone equipment, telephoneconferencing equipment, video conference equipment, telephones, cellulartelephones, smartphones, tablet computers, electronic mail applications,text message applications, thermostats, thermometers, motion sensors,physiological sensors, activity sensors, biomedical sensors orcombinations thereof.
 4. The system of claim 1, wherein the processingmodules comprise an assistive information detection module to use thecaptured data to identify assistive information.
 5. The system of claim4, wherein the assistive information detection module uses the captureddata in at least one of static searches and dynamic searches over fixeddatabases and continuous data streams.
 6. The system of claim 1,wherein: the captured data further comprise physiological data for oneor more participants; and the processing modules use the captured datato identify a physical or emotional condition of at least oneparticipant.
 7. The system of claim 1, wherein the assistive informationvalue and the intervention cost comprise a plurality of discretequantities based on a type of assistive information and a severity ofthe information cost on the dialog.
 8. The system of claim 1, wherein:the processing modules comprise an intervention delivery moduledetermine intervention parameters for communicating the assistiveinformation to the one or more participants; and the interface modulesintervene in the dialog in accordance with the intervention parameters.9. The system of claim 8, wherein: the intervention delivery moduleidentifies target participants in the dialog to receive the assistiveinformation; the processing modules comprise an intervention opportunityanalysis module to identify an appropriate time during the dialog forintervening in the dialog that minimizes interference with theparticipants; and the dialog environment comprises interventionmechanisms associated with the target participants and in communicationwith the interface modules that can be used to deliver the assistiveinformation to the target participants.
 10. The system of claim 9,wherein the intervention mechanisms comprise speakers, video monitors,lights, haptic systems, kinesthetic systems, graphical user interfaces,smart phones, tablet computers, telephones, cellular telephones orcombinations thereof.
 11. The system of claim 1, wherein the processingmodules comprise an intervention result analysis module to determinepost-intervention impacts on the dialog resulting from intervening inthe dialog and delivering the assistive information and to use thepost-intervention impacts to determine an intervention index fordelivering additional assistive information to one or more participantsat a later time during the dialog.
 12. The system of claim 11, whereinthe intervention result analysis module analyzes data captured from thedialog and participants after intervening in the dialog to determine thepost-intervention impacts of intervening in the dialog to deliver theassistive information.
 13. The system of claim 11, wherein theintervention result analysis module associates the post-interventionimpacts with at least one of the dialog, a portion of the dialog, a typeof dialog, the assistive information, a category of assistiveinformation, a delivery mechanism used to communicate the assistiveinformation and one or more participants and stores thepost-intervention impacts and associations for use in determining theintervention index for delivering additional assistive information toone or more participants at the later time during the dialog.
 14. Adialog intervention system comprising: at least one dialog environmentcontaining at least two participants engaged in a dialog; and acognitive system in communication with the at least one dialogenvironment to monitor the dialog between the at least two participants,the cognitive system comprising: interface capture modules to capturedata from a dialog environment containing at least one of theparticipants, the captured data accurately capturing a context of thedialog and comprising content of the dialog; processing modules incommunication with the interface capture modules, the processing modulescomprising: an assistive information detection module to identifyingassistive information valuable to the context of the dialog; and anassistive information value and intervention cost analysis module to usethe captured data to determine an intervention index on an interactionflow among the participants from delivering the assistive information toone or more participants during the dialog.
 15. The system of claim 14,wherein: the assistive information value and intervention cost analysismodule compares the value of the assistive information to theintervention cost on the dialog; and the cognitive system comprisesinterface presentation modules to intervene in the dialog if the valueof the assistive information is sufficient to justify the interventioncost on the dialog.
 16. The system of claim 14, wherein the assistiveinformation value and intervention cost analysis module: assigns a lowvalue to assistive information comprising a system-generated suggestion;assigns a high value to assistive information comprising a resolution toa conflict occurring among the participants during the dialog; andassigns a medium value between the low value and the high value toassistive information comprising an answer to a participant generatedquestion.
 17. The system of claim 14, wherein the assistive informationvalue and intervention cost analysis module: assigns a low interventioncost level when intervening in the dialog elicits a positive reactionfrom one or more participants; assigns a high intervention cost levelwhen intervening in the dialog elicits a negative reaction from one ormore participants; and assigns a medium intervention cost level betweenthe low intervention cost level and the high intervention cost levelwhen intervening in the dialog elicits a neutral reaction from one ormore participants.
 18. The system of claim 15, wherein the processingmodules comprise an intervention result analysis module to determinepost-intervention impacts on the dialog resulting from intervening inthe dialog and delivering the assistive information and to use thepost-intervention impacts to determine an intervention index fordelivering additional assistive information to one or more participantsat a later time during the dialog.
 19. The system of claim 18, whereinthe intervention result analysis module associates the post-interventionimpacts with at least one of the dialog, a portion of the dialog, a typeof dialog, the assistive information, a category of assistiveinformation, a delivery mechanism used to communicate the assistiveinformation and one or more participants and stores thepost-intervention impacts and associations for use in determining theintervention index for delivering additional assistive information toone or more participants at the later time during the dialog.